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I. INTRODUCTION 

This Appeal is from a Final Office Action mailed on May 28, 2003 (Paper No. 16), finally 
rejecting claims 1-25 of the above-identified application, and an Advisory Action mailed on 
August 22, 2003 (Paper No. 1 8), Applicants commenced this Appeal by a Notice of Appeal that 
was filed on September 8, 2003, and hereby submit this Appeal Brief 
n. REAL PARTY IN INTEREST 

The real party in interest for the above-identified application is International Business 
Machines (IBM) Corporation, the assignee of the entire right, title and interest in and to the 
subject application by virtue of an assignment of record in the U.S. Patent and Trademark Office. 

III. RELATED APPEALS AND INTERFERENCES 

There are no Appeals or Interferences known to Applicant, Applicant's representatives or 
the Assignee, which would directly affect or be indirectly affected by or have a bearing on the 
Board's decision in the pending Appeal. 

IV. STATUS OF CLAIMS 

Claims 1-25 are pending, stand rejected and are under appeal. The claims on appeal are 
set forth in the attached Appendix. 

Claims 1,10 and 19 are independent claims. Claims 2-9 depend directly or indirectly 
from claim 1. Claims 11-18 depend directly or indirectly from claim 10. Claims 20-25 depend 
directly or indirectly from claim 19. 

V. STATUS OF AMENDMENTS 

No after final Amendments were filed in this case subsequent to the Final Office Action 
of May 28, 2003 (Paper No. 16). 
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VI. SUMMARY OF THE INVENTION 

In general, the claimed inventions are directed to dialog management systems and 
methods for conversational computer systems with multiple input modalities. More specifically, 
the present invention relates to the management of multiple applications and input modalities 
through a conversational system, wherein many applications can be active at any time and the 
intended object of a user's action or command (dialog focus) can be deduced via focus 
resolution through an examination of the context of the user*s command. The command may be 
entered through any one of the several input modalities, examples of which include a spoken 
input, a keyboard input, a mouse input, etc. A detailed multi-modal history of events is 
maintained for, e.g., system events or commands that a user has previously performed and the 
final resolution of dialog focus will proceed through knowledge of any application specific 
aspects of the command and an investigation of the multimodal history. 

More specifically, the inventions of claims 1,10 and 19 will be further discussed with 
detail to the exemplary diagrams 1-4 of Applicants' Specification for purposes of illustration, but 
nothing herein shall be construed as placing any limitation on the claimed inventions. Claim 20 is 
directed to a system for determining and maintaining dialog focus in a conversational speech 
system. As depicted in the exemplary embodiment of Fig. 1 of Applicants' specification, the 
system (8) comprises a dialog manager (14) that is adapted to receive commands (12) fi-om a 
user. As shown in Fig. 3, the dialog manager (14) maintains a current dialog focus (32) for one 
of a plurality of active applications and a list of expected responses (34) for determining a current 
context of the commands received. 

The system (8) further comprises a multi-modal history (1 6) coupled to the dialog 
manager (1 4). The multi-modal history (16) provides a mechanism for capturing and 
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maintaining a complete history of all events concerning the system (8), and plays a role in 
deducing a target (1 8) of a user spoken command or response (12). As depicted in the 
exemplary embodiment of Fig. 2, the multi -modal history (16) maintains an event list of all 
events which affected a state of the system (8). The multi -modal history (16) is adapted to 
provide input to the dialog manager (14) for determining the current context of the commands 
(12) received. As depicted in Fig. 2, the events in the multi-modal history (16) include change of 
dialog focus events, for example. 

The system (8) further comprises a control component (20) which is adapted to select at 
least one method that is responsive to the commands received such that the system (8) applies 
methods responsive to the commands for an appropriate application including one of the active 
applications. Further, as specifically recited in claims 1 and 10, the application associated with 
the command is one of a plurality of active applications that are unknown to the dialog manager 
(14). In other words, by way of example, as explained on page 14, lines 1-14 of Applicants' 
specification, the component control (20) maintains a reference to all currently active 
applications. The dialog manager (14) may, for example, determine that the target (1 8) 
application is a calendar application, but the dialog manager (14) has no knowledge of which 
particular application implements a calendar. This degree of abstraction permits a suite of 
applications currently active to be modified dynamically, at the user's discretion, with no 
modification to the dialog manager (14) required. 



3 



Vn. ISSUES AND REJECTIONS 

Pending claims 1-25 stand rejected under 35 U.S.C. § 103(a) as being unpatentable over 
U.S. Patent No. 5,748,974 to Johnson in view of U.S. Patent No. 5,805,775 to Eberman et al, and 
in further view of U.S. Patent No. 5,265,014 to Haddock. 

Thus, the issue on appeal is whether the combination of Johnson . Eberman and Haddock 
is legally sufficient to establish a prima facie case of obviousness against claims 1-25. 

Vin. GROUPING OF CLAIMS 

Each Claim 1-25 stands or falls by itself 

IX. ARGUMENTS 

A. The Combination of Johnson , Eberman and 

Haddock is Legally Deficient to Establish a Prima Facie Case 
Of Obviousness Against the Claimed Inventions 

In rejecting claims under 35 U.S.C. 103, the Examiner bears the initial burden of 

presenting a prima facie case of obviousness. In re Riickaert, 9 F.3d 1 53 1 , 1 532 (Fed. Cir. 1 993). 

The burden of presenting a prima facie case of obviousness is only satisfied by showing some 

objective teaching in the prior art or that knowledge generally available to one of ordinary skill in 

the art would lead that individual to combine the relevant teachings of the references. In re Fine , 

837 F.2d 1071, 1 074 (Fed. Cir. 1988). A prima facie case of obviousness is established when 

the teachings of the prior art itself would appear to have suggested the claimed subject matter to 

one of ordinary skill in the art. In re Bell 991 F.2d 781, 782 (Fed. Cir. 1993). The suggestion to 




combine the references should come from the prior art, and the Examiner cannot use hindsight 
gleaned from the invention itself to pick and choose among related disclosures in the prior art to 
arrive at the claimed invention. In re Fine. 837 F.2d at 1075. If the Examiner fails to establish a 
prima facie case, the rejection is improper and must be overturned. In re Riickaert . 9 F.3d at 1532 
(citing In re Fine. 837 F.2d at 1074). 

Here, it is respectfully submitted that at the very minimum, the combination of Johnson . 
Eberman and Haddock is legally deficient to establish a prima facie case of obviousness under 35 
U.S.C. 1 03 to support the rejection of claims 1,10 and 1 9. Indeed, it is respectfully submitted 
that the obviousness rejections are based on impermissible hindsight reasoning through selective 
teachings of Johnson. Eberman and Haddock , and that no motivation has been shown for 
justifying the combination of such teachings as against claims 1,10 and 19. Indeed, at the very 
minimum. Applicants respectfully submit that such combination does not teach or suggest a 
dialog manager using a multi-modal history for determining the current context of a command 
received, wherein the command is associated with one of a plurality of active applications, as 
essentially claimed in claims 1, 10 and 19, much less wherein the events in the multi-modal 
history include change of dialog focus events, as essentially claimed in claim 19. 

In the Final Office Action, Examiner acknowledges that Johnson does not disclose the 
application associated with the command being one of a plurality of active applications. In an 
attempt to cure the deficiencies of Johnson , Examiner cites Eberman as disclosing an application 
user interface which allows the user to control the activities of multiple applications ... 
Examiner then presumably acknowledges that neither Johnson nor Eberman discloses the dialog 
manager determining current context of the command by reviewing a multi-modal history of 
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events^ but relies on Haddock as disclosing a multi-modal user interface for removing referential 
ambiguity from a natural language input. 

Although Eberman arguably discloses multiple applications, Eberman does not disclose 
or suggest a dialog manager using a multi-modal history for determining the current context of 
a command received, wherein the command is associated with one of a plurality of active 
applications^ In fact, Eberman discloses a mechanism wherein each application registers a set of 
rules, wherein portions of input texts that are recognized as corresponding to text of the rules are 
used to determine which commands for a given application to execute (see, e.g., Col. 5, lines 60- 
65). Thus, the Eberman system is limited in that the current context of a command received is 
determined by matching input text of a user to text of the registered. Eberman does not disclose 
a mechanism for resolving ambiguities, and input commands must m atch the texts of the rules. 

Furthermore, Haddock does not cure the deficiencies of Johnson and Eberman . In fact, 
the Examiner has not provided sufficient motivation for combining the teachings of Haddock 
with the teaching of Johnson and Eberman to support the contention that such combination 
teaches using a multi-modal history to determining the current context of a command received, 
wherein the command is associated with one of a plurality of active applications, as essentially 
claimed in claims 1,10 and 19. 

On page 4 of the Office Action, it is contended that "it would have been obvious to one of 
ordinary skill at the time of the invention to modify Johnso n to implement determining current 
context of user input response by reviewing a history of events as suggested by Haddock, for the 
purpose of providing a user friendly way to resolve ambiguity in a natural language system." It is 
respectfully submitted that such basis for obviousness is legally deficient and misses the point. 



6 



To begin. Haddock clearly does not cure the deficiencies of Johnson and Eberman with 
regard to disclosing or suggesting a dialog manager using a muUi-modal history to determining 
the current context of a command received, wherein the command is associated with one of a 
plurality of active applications^ as essentially claimed in claims 1,10 and 1 9. Haddock discloses 
nothing more than a user interface that enables natural language communication between a user 
and a computer database (single application, not multiple applications) by providing a 
mechanism for a user to remove a referential ambiguity from a database query by pointing to a 
textual reference on a visixal display (see, e.g.. Col. 2, lines 50-58). For example, as depicted in 
Fig. 2, a display screen has a "history" portion which displays previous questions and responses 
with respect to user database queries, allowing the user to point to any number of elements in one 
or more previous responses to which he/she intends to refer, to thereby provide a referential input 
along for a next database query (see, e.g.. Col. 3, lines 1-30). 

Those of ordinary skill in the art would readily appreciate that the "history" of previous 
questions or response for database queries as disclosed by Haddock is not remotely equivalent to 
a "multi-modal history" of events a multi-modal history that maintains an "event list" of all 
events which affected a state of the system, wherein the events in the multi-modal history include 
change of dialog focus events, as claimed in claim 10 for example. In fact, since Haddock does 
not disclose or suggest multiple active applications, the "history" of Haddock clearly does not 
include "change of dialog focus events", for example. 

In particular, by way of example, even assuming arguendo, that the query cards disclosed 
by Haddock comprises a "history of events" as contended by Examiner, the Haddock system 
relies on manual user interaction with such "history" and actual human intelligence processing on 
part of the user to resolve an ambiguity of a command. This has no relation to the claimed 
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process of determining by a dialog manager the current context of a command by reviewing a 
history of events, wherein the current command is associated with one of a plurality of active 
a pplication, as essentially claimed in claims K 10 and 19 

Furthermore, Haddock is limited to a single active application (a database application). 
As such, Haddock does not disclose or remotely suggest method for determining current context 
of commands between multiple active applications. Li fact. Haddock provides no teaching of any 
mechanism for using a multi-modal history to determining the current context of a command 
received, wherein the command is associated with one of a plurality of active applications^ as 
essentially claimed in claims 1,10 and 19. 

Furthermore, with respect to claim 1 9, given that Haddock discloses user interaction with 
a single database application. Examiner cannot reasonably contend that Haddock teaches or even 
remotely suggests wherein the events in the multi-modal history include change of dialog focus 
events, as essentially claimed in claim 19. Indeed, although Haddock arguably discloses 
maintaining previous queries, there is nothing in Haddock that remotely suggests a history of 
events including change of dialog focus events - indeed, with one active application, there is 
never a change in dialog focus to a different application. 

Furthermore, v^th respect to claims 4, 5,13,1 4, 22 and 23, for example, the cited 
combination of references clearly does not disclose or suggest wherein the multi-modal history 
includes a linked list of all events, much less wherein the events in the multi-modal history 
include at least one of events linked by time, by type, by transaction, by class and by dialog 
focus. Indeed, as depicted in Fig. 2 of Applicants' Specification, a linked list of events (22) 
includes events that are linked by time, event type (28), transaction identifiers, and event class 
(30). As shown in Fig. 1 , a link manager (1 5) in the multi-modal history (15) returns events of 
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certain types depending on context. Examiner's reliance on Haddock (see, page 6 of the Final 
Office Action, for example) as disclosing these claims features is clearly misplaced. Again, 
although Haddock may maintain a "history" of previous queries. Haddock does not teach or 
suggest that such "events" are "linked" as contemplated by the invention, hi fact, in the Haddock 
system, because there is no automated context resolution, there is no need for linking "events" 
because they merely have to be "displayed" for manual selection by the user. 

Thus, for at least the above reasons, claims 1,10 and 19 are believed to be patentable and 
non-obvious over the combination of Johnson, Eberman and Haddock. Further, all dependent 
claims are believed to be nonobvious and patentable over such combination at least for the 
reasons given above for respective base claims 1,10 and 19. 

Accordingly, reversal of the claim rejections under 35 U.S.C. 103(a) is respectfully 
requested. 

B. CONCLUSION 

Accordingly, it is respectfully requested that the Board reverse the claim rejections under 
35 U.S.C. § 103(a). 
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APPENDIX A 

1 . A method for determining and maintaining dialog focus in a conversational 
speech system comprising the steps of: 

presenting a command associated with an application to a dialog manager, the application 
associated with the command being one of a plurality of active applications that are unknown to 
the dialog manager; 

the dialog manager determining a current context of the command by reviev^ng a multi- 
modal history of events; 

determining at least one method responsive to the command based on the current context; 

executing the at least one method responsive to the command associated with the 
application; and 

changing dialog focus to one of the active applications based on user interaction with the 
conversational system. 

2. The method as recited in claim 1 , wherein the step of presenting a command 
includes the step of employing at least one multi-modal device for presenting the command. 

3. The method as recited in claim 2, wherein the at least one multi-modal device for 
presenting the command includes one of a telephone, a computer, and a personal digital assistant. 

4. The method as recited in claim 1 , wherein the step of determining a current 
context of the command by reviewing a multi-modal history of events includes the step of 
providing a linked list of all events in the multi-modal history. 

5. The method as recited in claim 4, wherein the events in the multi-modal history 
includes at least one of events linked by time, by type, by transaction, by class and by dialog 
focus. 

6. The method as recited in claim 1 , wherein the step of determining at least one 
method includes the step of referencing all active applications using a component control to 
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determine the at least one method which is appropriate based on the current context of the 
command. 

7. The method as recited in claim 1 , wherein the command is presented in a formal 
language such that a plurahty of hxmian utterances represent an action to be taken. 

8. The method as recited in claim 1 , wherein the step of determining a current 
context of the command by reviewing a multi-modal history of events includes the step of 
maintaining a current dialog focus and a list of expected responses in the dialog manager to 
provide a reference for determining the current context. 

9. The method as recited in claim 1 , further comprising the step of querying a user 
for one of information needed to resolve the current context and information needed to take an 
appropriate action, 

1 0. A program storage device readable by machine, tangibly embodying a program of 
instructions executable by the machine to perform method steps for determining and maintaining 
dialog focus in a conversational speech system, the method steps comprising: 

presenting a command associated with an application to a dialog manager, the application 
associated with the command being one of a plurality of active applications that are unknown to 
the dialog manager; 

the dialog manager determining a current context of the command by reviewing a multi- 
modal history of events; 

determining at least one method responsive to the command based on the current context; 

executing the at least one method responsive to the command associated with the 
application; and 

changing dialog focus to one of the active applications based on user interaction with the 
conversational system. 



11 



1 1 . The program storage device as recited in claim 1 0, wherein the step of presenting 
a command includes the step of employing at least one multi-modal device for presenting the 
command. 

1 2. The program storage device as recited in claim 1 1 , wherein the at least one multi- 
modal device for presenting the command includes one of a telephone, a computer, and a 
personal digital assistant. 

1 3. The program storage device as recited in claim 1 0, wherein the step of 
determining a current context of the command by reviewing a multi-modal history of events 
includes the step of providing a linked list of all events in the multi-modal history. 

1 4. The program storage device as recited in claim 1 3, wherein the events in the 
multi-modal history include at least one of events linked by time, by type, by transaction, by class 
and by dialog focus. 

1 5. The program storage device as recited in claim 1 0, wherein the step of 
determining at least one method includes the step of referencing all active applications using a 
component control to determine the at least one method which is appropriate based on the current 
context of the command. 

1 6. The program storage device as recited in claim 1 0, wherein the command is 
presented in a formal language such that a plurality of human utterances represent an action to be 
taken. 

1 7. The program storage device as recited in claim 1 0, wherein the step of 
determining a current context of the command by reviewing a multi-modal history of events 
includes the step of maintaining a dialog focus and a list of expected responses in the dialog 
manager to provide a reference for determining the current context. 
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1 8. The program storage device as recited in claim 1 0, further comprising the step of 
querying a user for one of information needed to resolve the current context and information 
needed to take an appropriate action. 

1 9. A system for determining and maintaining dialog focus in a conversational speech 
system comprising: 

a dialog manager adapted to receive commands from a user, the dialog manager 
maintaining a current dialog focus for one of a plurality of active applications and a list of 
expected responses for determining a current context of the commands received; 

a multi-modal history coupled to the dialog manager for maintaining an event list of all 
events which affected a state of the system, the multi-modal history adapted to provide input to 
the dialog manager for determining the current context of the commands received, wherein the 
events in the multi-modal history include change of dialog focus events; and 

a control component adapted to select at least one method responsive to the commands 
received such that the system applies methods responsive to the commands for an appropriate 
application including one of the active applications. 

20. The system as recited in claim 1 9, wherein the appropriate application further 
includes one of an inactive application, an application with a graphical component and an 
application with other than a graphical component. 

21 . The system as recited in claim 1 9, wherein the commands are input to the dialog 
manager by one of a telephone, a computer, and a personal digital assistant. 

22. The system as recited in claim 1 9, wherein the multi-modal history includes a 
linked list of all events to associate a given command to the appropriate application. 

23. The system as recited in claim 22, wherein the events in the multi-modal history 
include at least one of events linked by time, by type, by transaction, by class and by dialog 
focus. 
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24, The system as recited in claim 1 9, wherein the control component references all 
active applications to determine the at least one method which is appropriate based on the current 
context of the commands. 

25. The system as recited in claim 1 9, wherein the command is presented in a formal 
language such that a plurality of human utterances represent an action to be taken. 
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